Apparatus and method for performing color transformation on raw sensor images

ABSTRACT

A method for processing image data may include: obtaining a raw input image that is captured under an input illumination; obtaining a target illumination from a user input; obtaining an intermediate image having colors captured under a reference illumination, from the raw input image, based on a first color transform that maps the input illumination to the reference illumination in an illumination dataset of raw sensor images that are captured under a plurality of different illuminations; and obtaining an output image having colors captured under the target illumination, from the intermediate image, based on a second color transform that maps the reference illumination in the illumination dataset to the target illumination.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to U.S. Provisional Patent Application No. 63/390,903 filed on Jul. 20,2022, in the U.S. Patent & Trademark Office, the disclosure of which isincorporated herein by reference in its entirety.

BACKGROUND 1. Field

The disclosure relates to an apparatus and a method for convertingcolors of an input sensor image that is captured under any arbitraryinput illumination to colors under any arbitrary target illumination,such that an output image appears as if it had been captured under thetarget illumination.

2. Description of Related Art

Capturing raw sensor images under various settings is quite challenging,especially under various illuminations and lighting conditions. Theprocess requires adjusting camera settings, using tripods, setting upthe scene, and likely finding different lighting conditions andenvironments. With such limitations, it is effort and time consuming tocapture large scale datasets of raw sensor images for training neuralnetwork models.

For example, a mobile device is updated with a new image sensor, newtraining datasets need to be recaptured because the new image sensor mayhave different characteristics (e.g., spectral sensitivity, noiseprofile, etc.). The time and effort needed to capture training data is asignificant challenge to the manufacturer of the mobile device and othersmartphone and camera companies.

Therefore, there has been a demand for a method for augmenting existingdatasets of raw sensor images to obtain synthetic images with variousillumination conditions. However, in previous color transformationmethods, color mapping is inaccurate and could only be performed for aspecific set of illuminations rather than being capable of convertingimages between arbitrary illuminations.

SUMMARY

One or more embodiments of the present disclosure provide an apparatusand method for converting colors of an input sensor image that iscaptured under any arbitrary input illumination to colors under anyarbitrary target illumination such that an output image appears as if ithad been captured under the target illumination.

Further, one or more embodiments of the present disclosure provide anapparatus and method for performing data augmentation for trainingneural networks by converting colors of an input sensor image.

Still further, one or more embodiments of the present disclosure providean apparatus and method for providing photo editing options forconverting colors of a raw input image from an arbitrary inputillumination to an arbitrary target illumination.

According to an aspect of the present disclosure, an electronic devicefor processing image data, may include: a user interface configured toreceive a target illumination; at least memory storing instructions andconfigured to store an illumination dataset of raw sensor images thatare captured under a plurality of different illuminations including areference illumination; and at least one processor configured to executethe instructions to: obtain a raw input image that is captured under aninput illumination; obtain an intermediate image having colors capturedunder the reference illumination, from the raw input image, based on afirst color transform that maps the input illumination to the referenceillumination in the illumination dataset; and obtain an output imagehaving colors captured under the target illumination, from theintermediate image, based on a second color transform that maps thereference illumination in the illumination dataset, to the targetillumination.

The at least one processor may be further configured to performillumination estimation on the raw input image that is transformed bythe first color transform and the second color transform, via anauto-white-balance module.

The at least one processor may be further configured to create anaugmented data set including the output image, and input the augmenteddata set to an artificial intelligence (AI)-based image processing modelto train the AI-based image processing model.

The raw sensor images and the raw input image may be unprocessed Bayerimages or images which are not corrected via an image signal processor(ISP).

The first color transform and the second color transform may berepresented as a first transformation matrix and a second transformationmatrix, respectively, wherein the at least one processor may be furtherconfigured to: obtain the first color transform via a first neuralnetwork configured to receive a ratio of RGB values of the inputillumination and output elements of the first transformation matrix; andobtain the second color transform via a second neural network configuredto receive a ratio of RGB values of the target illumination and outputelements of the second transformation matrix.

The first neural network and the second neural network may be trainedusing a training input illumination, a training target illumination, andthe reference illumination that are obtained from the illuminationdataset, wherein the training input illumination, the training targetillumination, and the reference illumination may be obtained from anachromatic patch in a color rendition chart of the raw sensor images.

The at least one processor may be further configured to: based on theillumination dataset not including the input illumination, identify a Knumber of illuminations that are nearest to the input illumination inthe illumination dataset, and use a weighted sum of color transforms ofthe K number of illuminations as the first color transform, wherein Kdenotes a natural number that is greater than or equal to 2.

The at least one processor may be further configured to: based on theillumination dataset not including the target illumination, identify a Knumber of illuminations that are nearest to the target illumination inthe illumination dataset, and use a weighted sum of color transforms ofthe K number of illuminations as the second color transform, wherein Kdenotes a natural number that is greater than or equal to 2.

The electronic device may further include: a camera configured tocapture the raw input image; and a display, wherein the user interfaceis further configured to receive, as a user input, the targetillumination and a request for creating a synthesized image of the rawinput image, and wherein the at least one processor may be furtherconfigured to control the display to display the output image as thesynthesized image of the raw input image that is re-illuminated underthe target illumination.

According to another aspect of the present disclosure, a method forprocessing image data, may include: obtaining a raw input image that iscaptured under an input illumination; obtaining a target illuminationfrom a user input; obtaining an intermediate image having colorscaptured under a reference illumination, from the raw input image, basedon a first color transform that maps the input illumination to thereference illumination in an illumination dataset of raw sensor imagesthat are captured under a plurality of different illuminations; andobtaining an output image having colors captured under the targetillumination, from the intermediate image, based on a second colortransform that maps the reference illumination in the illuminationdatabase, to the target illumination.

The method may further include performing illumination estimation on theraw input image transformed by the first color transform and the secondcolor transform, via an auto-white-balance module.

The method may further include: creating an augmented data set includingthe output image, and input the augmented data set to an artificialintelligence (AI)-based image processing model to train the AI-basedimage processing model.

The raw input image may be an unprocessed Bayer image or an image whichis not corrected via an image signal processor (ISP).

The first color transform and the second color transform may berepresented as a first transformation matrix and a second transformationmatrix, respectively. The method may further include: obtaining thefirst color transform via a first neural network configured to receive aratio of RGB values of the input illumination and output elements of thefirst transformation matrix; and obtaining the second color transformvia a second neural network configured to receive a ratio of RGB valuesof the target illumination and output elements of the secondtransformation matrix.

The method may further include: training the first neural network usinga training input illumination and the reference illumination; andtraining the second neural network using the reference illumination anda training target illumination, wherein the training input illumination,the training target illumination, and the reference illumination areobtained from the illumination dataset.

The method may further include: based on the illumination dataset notincluding the input illumination, identifying a K number ofilluminations that are nearest to the input illumination in theillumination dataset, and using a weighted sum of color transforms ofthe K number of illuminations as the first color transform, wherein Kdenotes a natural number that is greater than or equal to 2.

The method may further include: based on the illumination dataset notincluding the target illumination, identifying a K number ofilluminations that are nearest to the target illumination in theillumination dataset, and using a weighted sum of color transforms ofthe K number of illuminations as the second color transform, wherein Kdenotes a natural number that is greater than or equal to 2.

According to another aspect of the present disclosure, there is provideda non-transitory computer-readable storage medium storing a program thatis executable by at least one processor to perform the method forprocessing the image data.

Additional aspects will be set forth in part in the description thatfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and aspects of embodiments of thedisclosure will be more apparent from the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a diagram of an electronic device for performing colortransformation according to embodiments of the present disclosure;

FIG. 2 illustrates a color transformation method according toembodiments of the present disclosure;

FIG. 3 illustrates a method of collecting an illumination data set forestimating a color transform according to embodiments of the presentdisclosure;

FIG. 4 illustrates a method of indexing images to provide anillumination dataset, according to embodiments of the presentdisclosure;

FIG. 5 illustrates a method of training a first neural network and asecond neural network, and a method of predicting color transforms usingthe trained first neural network and the trained second neural network,according to embodiments of the present disclosure;

FIGS. 6A and 6B illustrate structures of neural networks configured toestimate color transforms according to embodiments of the presentdisclosure;

FIG. 7A illustrates a method of obtaining a first color transform thatconverts an arbitrary input illumination to a reference illumination,and FIG. 7B illustrates a method of obtaining a second color transformthat converts a reference illumination to an arbitrary targetillumination, according to embodiments of the present disclosure;

FIG. 8A illustrates a method of obtaining a first color transform thatconverts an arbitrary input illumination to a reference illuminationwhen the arbitrary input illumination does not exist in an illuminationdataset, and FIG. 8B illustrates a method of obtaining a second colortransform that converts a reference illumination to an arbitrary targetillumination when the arbitrary target illumination does not exist in anillumination dataset;

FIG. 9 illustrates a method of applying a first color transform and asecond color transform to an input image according to embodiments of thepresent disclosure; and

FIG. 10 illustrates a result of converting an input illumination to areference illumination and converting the reference illumination to atarget illumination according to embodiments of the present disclosure;

FIG. 11 illustrates a method of converting an input image under an inputillumination directly to an output image under a target illuminationwithout using a middle proxy, such as a daylight illumination;

FIG. 12 illustrates a method of training a neural network-basedtransform estimator according to embodiments of the present disclosure;

FIG. 13 illustrates a use application of a color transformation methodaccording to embodiments of the present disclosure;

FIG. 14 is a diagram of devices for performing color transformationaccording to embodiments of the present disclosure; and

FIG. 15 is a diagram of components of one or more devices of FIG. 14according to embodiments of the present disclosure.

DETAILED DESCRIPTION

Example embodiments are described in greater detail below with referenceto the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exampleembodiments. However, it is apparent that the example embodiments can bepracticed without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure the description with unnecessary detail.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list. For example, the expression, “at leastone of a, b, and c,” should be understood as including only a, only b,only c, both a and b, both a and c, both b and c, all of a, b, and c, orany variations of the aforementioned examples.

While such terms as “first,” “second,” etc., may be used to describevarious elements, such elements must not be limited to the above terms.The above terms may be used only to distinguish one element fromanother.

The term “module” or “component” is intended to be broadly construed ashardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of possible implementations. In fact,many of these features may be combined in ways not specifically recitedin the claims and/or disclosed in the specification. Although eachdependent claim listed below may directly depend on only one claim, thedisclosure of possible implementations includes each dependent claim incombination with every other claim in the claim set.

No element, act, or instruction used herein should be construed ascritical or essential unless explicitly described as such. Also, as usedherein, the articles “a” and “an” are intended to include one or moreitems, and may be used interchangeably with “one or more.” Furthermore,as used herein, the term “set” is intended to include one or more items(e.g., related items, unrelated items, a combination of related andunrelated items, etc.), and may be used interchangeably with “one ormore.” Where only one item is intended, the term “one” or similarlanguage is used. Also, as used herein, the terms “has,” “have,”“having,” or the like are intended to be open-ended terms. Further, thephrase “based on” is intended to mean “based, at least in part, on”unless explicitly stated otherwise.

One or more embodiments of the present disclosure provide performing anillumination synthesis process that includes: estimating a colortransform that maps an input illumination applied to an input image to atarget illumination; and applying the color transform to the input imageto obtain an output image under the target illumination.

The input sensor image may be an unprocessed raw image that is outputfrom an image sensor. The term “raw image” may refer to an unprocesseddigital output of an image sensor of a camera, and may be also referredto as a “raw burst image” or “Bayer image.” Light or photons incidentfrom a scene are digitalized and recorded by a camera sensor, and theraw image is constituted with digital pixel intensity values recorded bythe camera sensor before any processing is applied. For example, the rawimage is an image that is not processed via an image signal processor(ISP) or an image processing unit (IPU), and may have a raw Bayerformat. When the camera sensor includes sensor elements that arearranged in a pattern of red, green, and blue color channels, which iscalled a Bayer array, an image recorded by the Bayer array on the camerasensor is called the Bayer image.

Further, one or more embodiments of the present disclosure providemapping colors of an input sensor image that is captured under anyarbitrary illumination, to any arbitrary target illumination such thatan output image appears as if it was captured under the targetillumination.

Still further, one or more embodiments of the present disclosure providean apparatus and a method for estimating a transform matrix thatconverts the colors of the input sensor image that is captured under aninput illumination, to colors that are obtained under a targetillumination.

Still further, one or more embodiments of the present disclosure providean apparatus and a method for applying the transform matrix tooriginally captured images to perform a white-balance correction, or togenerate synthetic images for data augmentation.

FIG. 1 is a diagram of an electronic device for performing colortransformation according to embodiments of the present disclosure.

As shown in FIG. 1 , the electronic device may include a camera 100, atleast one processor 200, at least one memory 300, an input interface400, a display 500, and a communication interface 600.

The camera 100 may include a lens 110, and an image sensor 120 that mayinclude a complementary metal oxide semiconductor (CMOS) sensor or acharge-coupled device (CCD) sensor, and a color filter array (CFA). Thecamera 100 may capture an image based on a user input that is receivedvia the input interface 400, and may output an unprocessed raw image tothe processor 200. The input interface 400 may be implemented as atleast one of a touch panel, a keyboard, a mouse, a button, a microphone,and the like.

The processor 200 may include an artificial intelligence (AI)-basedimage processing model that includes an auto-white-balance module 210, acolor transform estimation module 220, and a color transformation module230. While the auto-white-balance module 210 is illustrated as beingprovided outside the camera 100, the auto-white-balance module 210 maybe provided inside the camera 100 depending on embodiments. Theprocessor 200 may receive an input image from the camera 100, or from anexternal device via the communication interface 600. The input image maybe a raw image that is not processed by an image signal processor (ISP),and/or which has a Bayer format.

Based on a user input for editing the input image, for example, a userinput requesting color transformation to convert the input image from aninput illumination to a target illumination, the processor 200 mayperform color transformation on the input image, and output thecolor-transformed input image as a processed image. The processor 200may output the processed image via the display 500, and may store theprocessed image in the memory 300 as an augmented image or a synthesizedimage.

In embodiments of the disclosure, the processor 200 may estimate a colortransform that maps an input illumination of an input image to anyarbitrary target illumination, and may apply the color transform to theinput image to obtain an output image under the target illumination.

In detail, referring to the processor 200, the auto-white-balance module210 may identify an input illumination under which the input image iscaptured, and the color transform estimation module 220 may estimate afirst color transform that maps the input illumination to a referenceillumination (e.g., a daylight illumination), and a second colortransform that maps the reference illumination to a target illumination.

The color transform estimation module 220 may estimate a color transformusing a linear mapping method (e.g., y=ax+b where saturated pixels areexcluded during estimation of the color transform), a non-linear mappingmethod (e.g., a second degree polynomial having a form of ax²+bx+c orhigher order functions), or a neural network.

In an embodiment, either one or both of a first color transform and asecond color transform may be estimated using a linear mapping method.For example, a color transform T (e.g., the first color transform or thesecond color transform) between a first illumination L₁[r₁, g₁, b₁] anda second illumination L₂[r₂, g₂, b₂] may be obtained as follows:

$\begin{bmatrix}{r2/r1} & 0 & 0 \\0 & {g2/g1} & 0 \\0 & 0 & {b2/b1}\end{bmatrix}$

For example, for the first color transform, the first illuminationL₁[r₁, g₁, b₁] and the second illumination L₂[r₂, g₂, b₂] may correspondto an illumination of an input image and a reference illumination,respectively. For the second color transform, the first illuminationL₁[r₁, g₁, b₁] and the second illumination L₂[r₂, g₂, b₂] may correspondto the reference illumination and a target illumination, respectively.

More specifically, given a first image I₁∈R^(N×3) under the firstillumination L₁∈R³, and a second image I₂∈R^(N×3) under the secondillumination L₂∈R³ with n pixels of the same scene, a linear colortransform T∈R^(3×3) between the color values of the first image I₁ andthe second image I₂ may be expressed as I₂≈I₁T. T may be computed usinga pseudo inverse as follows: T=(I₁ ^(T)I₁)⁻¹I₁ ^(T)I₂.

For example, the linear color transform T may be represented in a 3×3color transform matrix as follows:

$T_{3 \times 3} = \begin{pmatrix}t_{1} & t_{2} & t_{3} \\t_{4} & t_{5} & t_{6} \\t_{7} & t_{8} & t_{9}\end{pmatrix}$

More specifically, given A denotes pixel values in R, G, B colorchannels for the first image I₁, B denotes pixel values in R, G, B colorchannels for the second image I₂, the 3×3 color transform matrix Tbetween A and B is calculated as follows.

${{A \times T} = B}{A = \begin{bmatrix}a_{1R} & a_{1G} & a_{1B} \\a_{2R} & a_{2G} & a_{2B} \\\ldots & \ldots & \ldots \\a_{NR} & a_{NG} & a_{NB}\end{bmatrix}}{T = \begin{bmatrix}{t1} & {t2} & {t3} \\{t4} & {t5} & {t6} \\{t7} & {t8} & {t9}\end{bmatrix}}{B = \begin{bmatrix}b_{1R} & b_{1G} & b_{1B} \\b_{2R} & b_{2G} & b_{2B} \\\ldots & \ldots & \ldots \\b_{NR} & b_{NG} & b_{NB}\end{bmatrix}}$

In the matrices of A and B, the three columns correspond to R, G, Bcolor channels, and the rows correspond to the number of pixels in theinput image I₁ and the output image I₂, respectively.

Using a pseudo-inverse equation, the 3×3 color transform matrix T iscalculated as follows:

$T = {{\left( {\begin{bmatrix}a_{1R} & a_{1G} & a_{1B} \\a_{2R} & a_{2G} & a_{2B} \\\ldots & \ldots & \ldots \\a_{NR} & a_{NG} & a_{NB}\end{bmatrix}^{T}\begin{bmatrix}a_{1R} & a_{1G} & a_{1B} \\a_{2R} & a_{2G} & a_{2B} \\\ldots & \ldots & \ldots \\a_{NR} & a_{NG} & a_{NB}\end{bmatrix}} \right)^{- 1}\begin{bmatrix}a_{1R} & a_{1G} & a_{1B} \\a_{2R} & a_{2G} & a_{2B} \\\ldots & \ldots & \ldots \\a_{NR} & a_{NG} & a_{NB}\end{bmatrix}}^{T}{\begin{bmatrix}b_{1R} & b_{1G} & b_{1B} \\b_{2R} & b_{2G} & b_{2B} \\\ldots & \ldots & \ldots \\b_{NR} & b_{NG} & b_{NB}\end{bmatrix}}}$

In the embodiment, the 3×3 color transform matrix is used since the 3×3color transform matrix is linear and accurate, and computationallyefficient. However, the size of the color transform matrix is notlimited thereto, and any 3×M color transform matrix (wherein 3) may beused.

In another embodiment of the present disclosure, the color transform Tmay be estimated using a neural network that is trained to estimate acolor transform. In particular, machine learning-based methods may beused to predict the color transforms that map between the inputillumination and the reference illumination, and between the referenceillumination and the target illumination. In estimating the colortransforms, a small dataset of images of color charts captured undervarious illuminations may be used without requiring a large trainingdataset. A method of estimating the color transform T using a neuralnetwork which will be described later with reference to FIGS. 5, 6A, and6B in detail.

The color transformation module 230 may apply the first color transformto an input image to obtain an intermediate image, and may apply thesecond color transform to the intermediate image to obtain an outputimage. The processor 200 is capable of mapping an image from anyarbitrary input illumination to any arbitrary target illumination, viathe two-step color transformation method using the referenceillumination (e.g., the daylight illumination) as a proxy. Using thedaylight illumination as a middle proxy may lead to accurate results dueto the fact that the daylight illumination has a wide spectraldistribution and under which most colors are well represented.

Additionally, the auto-white-balance module 210 may perform illuminationestimation on an input image transformed by the first color transformand the second color transform.

The color-transformed input image may be stored in the memory 300 as anaugmented image.

All the elements of the electronic device may be included in a singledevice, or may be included in more than one device. For example, thecamera 100, the input interface 400, and the display 500 may be includedin a client device (e.g., a smartphone), and the AI-based imageprocessing model of the processor 200 may be included in a server. Whenthe AI-based image processing model is included in the server, theclient device may send an input image and a target illumination to theserver, request the server to perform color transformation on the inputimage according to the target illumination to obtain a processed image,and may receive the processed image from the server.

FIG. 2 illustrates a color transformation method according toembodiments of the present disclosure.

According to an embodiment, a method of estimating a color transformthat maps an input image from an input illumination to a targetillumination, includes estimating a color transform that maps the inputillumination to a reference illumination (e.g., daylight illumination),and then another color transform that maps the reference illumination tothe target illumination, using the reference illumination as a middleproxy to improve the estimation accuracy of the color transform and toreduce a computational load.

In detail, as shown in FIG. 2 , the color transformation method mayinclude operation S211 of obtaining an input image captured under aninput illumination, operation 212 of estimating a first color transformthat converts an illumination condition of the input image from theinput illumination into a reference illumination, operation S213 ofapplying the first color transform to the input image, operation S241 ofobtaining the color-transformed input image as an intermediate image,operation S215 of estimating a second color transform that converts anillumination condition of the intermediate image into a targetillumination, operation 216 of applying the second color transform tothe intermediate image, and operation S217 of obtaining thecolor-transformed intermediate image as an output image.

In operation S211, the input image may be a raw sensor image that isoutput from the image sensor 120, without being processed via an imagesignal processor (ISP).

In operation S212, the input illumination of the input image may beidentified by an auto-white-balance (AWB) module that is included in thecamera 100 or the at least one processor 200. The first color transformmay be estimated using a linear or non-linear color transform estimationmethod, or a first neural network that is trained to receive, as input,the input illumination, and output the first color transform thatconverts the input image from the input illumination into the referenceillumination. A method of training the first neural network will bedescribed later with reference to FIGS. 3-5 .

In operations S213 and S214, the first color transform is applied to theinput image to re-illuminate the input image under the referenceillumination to obtain the intermediate image under the referenceillumination.

In operation S215, the second color transform may be estimated using alinear or non-linear color transform estimation method, or a secondneural network that is trained to receive, as input, the targetillumination, and output the second color transform that converts theintermediate image from the reference illumination to the targetillumination. A method of training the second neural network will bedescribed later with reference to FIGS. 3-5 .

In operations S216 and S217, the second color transform is applied tothe intermediate image to re-illuminate the intermediate image under thetarget illumination to obtain the output image under the targetillumination.

FIG. 3 illustrates a method of collecting an illumination data set forestimating a color transform according to embodiments of the presentdisclosure.

As shown in FIG. 3 , a light box 10 may be used to illuminate a colorchart 20 under different illuminations, such as for example, sunlight(or daylight), light emitting diode (LED) light, incandescent,fluorescent, and the like. Images of the color chart 20 may be capturedby a camera sensor 30 under the different illuminations. The images maybe used to train a neural network that estimates a color transform. Forexample, a first color chart image under sunlight, a second color chartimage under LED light, a third color charge image under incandescent,and a fourth color chart image under fluorescent may be captured andcollected as training data. Once images of the color chart are obtained,ground-truth illuminations may be estimated from the color chartscaptured in the images, using a neutral patch (also referred to as“achromatic patch” or “gray patch”) included in the color charts. Theground-truth illuminations may be stored in the memory 300 to be used intraining the first neural network and the second neural network, or tocompute color transforms via a linear or non-linear color transformestimation method. In FIG. 3 , the color chart is used as a color sourcefor providing diverse colors, but the embodiments of the presentdisclosure are not limited thereto, and diverse examples of colorfulmaterials may be used instead of the color chart.

FIG. 4 illustrates a method of indexing images to provide anillumination dataset, according to embodiments of the presentdisclosure. Raw sensor images that are captured under a plurality ofdifferent illuminations may be stored and indexed to provide anillumination dataset, and the illumination dataset may include areference illumination. The raw sensor images that are captured under aplurality of different illuminations may be referred to as color chartimages, and the color chart images may be obtained using the light box10.

As shown in FIG. 4 , each of the color chart images is indexed based ona scene illumination using a two-dimensional coordinate [R/G, B/G], andstored as an illumination dataset. For example, a first illumination L1[R1, G1, B1], a second illumination L2 [R2, G2, B2], a thirdillumination L3 [R3, G3, B3], and a fourth illumination L4 [R4, G4, B5]may be estimated from an RGB value of a neutral patch included in thefirst to the fourth color chart images. The first illumination L1 [R1,G1, B1], the second illumination L2 [R2, G2, B2], the third illuminationL3 [R3, G3, B3], and the fourth illumination L4 [R4, G4, B5] may beindexed using the two-dimensional coordinate [R/G, B/G], and stored asthe illumination dataset.

For each of the color chart images under the different illuminations,two color transforms between corresponding color chart values may beestimated. Specifically, a first color transform that maps an inputillumination to a reference illumination, and a second color transformthat maps the reference illumination to a target illumination, areestimated. A plurality of pairs of the first color transform and thesecond color transform may be obtained for the different illuminations,and may be used as a training dataset for training a neural network thatestimates a color transform between two arbitrary illuminations. Usingthis training data set, two color transform estimator models may betrained to predict color transforms corresponding to arbitraryilluminations. Among the two color transform estimator models, a firstmodel may predict a “from-arbitrary-to-reference” transform, and asecond model may predict a “from-reference-to-arbitrary” transform. Eachof the color transform estimator models may be implemented as amulti-layer perceptron (MLP). A color transform estimation process usingan artificial intelligence (AI)-based model will be further describedwith reference to FIGS. 5, 6A, and 6B.

Alternatively, the indexed illuminations of the color chart imagesillustrated in FIG. 4 may be used to compute the first color transformand the second color transform based on a mathematical algorithm withoutusing a neural network. A color transform estimation process withoutusing a neural network will be described later with reference to FIGS.7A, 7B, 8A, and 8B.

FIG. 5 illustrates a method of training a first neural network and asecond neural network, and a method of predicting color transforms usingthe trained first neural network and the trained second neural network,according to embodiments of the present disclosure.

Referring to FIG. 5 , a first neural network 40 is trained to receive,as input, an input illumination Lu, and output an estimated first colortransform that converts the input illumination Lu to a referenceillumination Ld. The reference illumination Ld may be a predeterminedknown value, such as a standard daylight illumination D65. A differencebetween the estimated first color transform and a ground-truth colortransform may be computed as a loss of the first neural network and theloss may be back-propagated to the first neural network 40 to updatenode weights of the first neural network 40. The ground-truth colortransform for the first neural network 40 may be obtained using theabove-discussed training dataset. For example, all colors in a colorchart that is captured in each of an input image and a reference imagemay be used to compute the ground-truth color transform T_(G1) for thefirst neural network 40, or when the input image and the reference imagedo not capture a color chart, any type of colorful objects orcalibration objects in the input image and the reference images may beused to compute the ground-truth color transform T_(G1). In other words,the ground-truth color transform T_(G1) is computed by comparingdifferent RGB values of an object that is observed under differentilluminations.

A second neural network 50 is trained to receive, as input, a targetillumination Lv, and output an estimated second color transform thatconverts the reference illumination Ld to the target illumination Lv. Adifference between the estimated second color transform and aground-truth color transform may be computed as a loss of the secondneural network 50, and the loss may be back-propagated to the secondneural network 50 to update node weights of the second neural network50. The ground-truth color transform for the second neural network maybe obtained using the above-discussed training dataset. All colors in acolor chart that is captured in each of the reference image and thetarget image may be used to compute the ground-truth color transformT_(G2) for the second neural network 50, or when the input image and thereference image do not capture a color chart, any type of colorfulobjects or calibration objects in the input image and the referenceimages may be used to compute the ground-truth color transform T_(G2).In other words, the ground-truth color transform T_(G2) is computed bycomparing different RGB values of an object that is observed underdifferent illuminations.

Once the first neural network 40 and the second neural network 50 aretrained, the first neural network 40 and the second neural network 50are run at an inference stage. At the inference stage, an inputillumination of an input image may be identified using anauto-white-balance (AWB) module, and a target illumination may beidentified from a user input. The input illumination is fed into thefirst neural network 40 to obtain, as output, a first color transformthat maps the input illumination to a reference illumination. The targetillumination is fed into the second neural network 50 to obtain, asoutput, a second color transform that maps the reference illumination tothe target illumination. The first color transform and the second colortransform may be provided as a matrix format (e.g., a 3-by-3 matrix).

FIGS. 6A and 6B illustrate structures of neural networks configured toestimate color transforms according to embodiments of the presentdisclosure.

As shown in FIG. 6A, the first neural network 40 may include an inputlayer, a set of hidden layers, and an output layer. For example, theinput layer may include two nodes to receive an input illumination Luexpressed as a two-dimensional coordinate [R/G, B/G]. For example, eachof the hidden layers and the output layer may include nine nodes, butthe number of the nodes is not limited thereto. The output layer mayoutput a first color transform T₁ that converts the input illuminationLu to a reference illumination Ld. In particular, the output layer mayoutput matrix elements (t₁₁, t₁₂, t₁₃, t₁₄, t₁₅, t₁₆, t₁₇, t₁₈, t₁₉) ofthe first color transform T₁, which may be converted into the followingmatrix:

$T_{1} = \begin{bmatrix}t_{11} & t_{12} & t_{13} \\t_{14} & t_{15} & t_{16} \\t_{17} & t_{18} & t_{19}\end{bmatrix}$

The second neural network 50 may have substantially the same structureas the first neural network 40. For example, as shown in FIG. 6B, thesecond neural network 50 may include an input layer, a set of hiddenlayers, and an output layer. For example, the input layer may includetwo nodes to receive a target illumination Lv expressed as atwo-dimensional coordinate [R/G, B/G]. Each of the hidden layers and theoutput layer may include nine nodes. The output layer may output asecond color transform T₂ that converts the reference illumination Ld tothe target illumination Lv. In particular, the output layer may outputmatrix elements (t₂₁, t₂₂, t₂₃, t₂₄, t₂₅, t₂₆, t₂₇, t₂₈, t₂₉) of asecond color transform T₂, which may be converted into the followingmatrix:

$T_{2} = \begin{bmatrix}t_{21} & t_{22} & t_{23} \\t_{24} & t_{25} & t_{26} \\t_{27} & t_{28} & t_{29}\end{bmatrix}$

As discussed above, the first color transform and the second colortransform may be obtained using artificial intelligence (AI)-basedmodels, but the embodiments of the present disclosure are not limitedthereto. For example, the first color transform and the second colortransform may be computed based on a mathematical algorithm using theillumination dataset collected as shown in FIG. 4 .

FIG. 7A illustrates a method of obtaining a first color transform thatconverts an arbitrary input illumination to a reference illumination,and FIG. 7B illustrates a method of obtaining a second color transformthat converts a reference illumination to an arbitrary targetillumination, according to embodiments of the present disclosure.Depending on embodiments of the present disclosure, a linear ornon-linear transform estimation method may be used as shown in FIGS. 7Aand 7B, in addition to or alternative to a neural-network basedtransform estimation method as shown in FIGS. 5 and 6 .

Referring to FIG. 7A, an illumination dataset including illuminationsL1, L2, L3, L4, LN of color chart images may be used to compute a firstcolor transform T₁₁, T₁₂, T₁₃, . . . T_(1N) that converts one of theinput illuminations L1, L3, L4, . . . , LN to a reference illuminationL2 where all colors (or a plurality of colors) in the color chart may beused to compute the first color transform. The reference illumination L2may be a predetermined illumination, such as a standard daylightillumination D65.

Referring to FIG. 7B, the illumination dataset including theilluminations L1, L2, L3, L4, . . . , LN of the color chart images maybe also used to compute a second color transform T₂₁, T₂₂, T₂₃, . . . ,T_(2N) that converts the reference illumination L2 to one of the othertarget illuminations L1, L3, L4, . . . , LN, where all colors (or aplurality of colors) in the color chart may be used to compute thesecond color transform.

The first color transform and the second color transform that areacquired via the methods shown in FIGS. 7A and 7B may be applied to araw input image example, by the color transformation module 230 of FIG.1 , or may be used as a first ground-truth color transform and a secondground-truth color transform to train the first neural network 40 andthe second neural network 50 of the color transformation estimationmodule 220 of FIG. 1 .

FIG. 8A illustrates a method of obtaining a first color transform thatconverts an arbitrary input illumination to a reference illuminationwhen the arbitrary input illumination does not exist in an illuminationdataset, and FIG. 8B illustrates a method of obtaining a second colortransform that converts a reference illumination to an arbitrary targetillumination when the arbitrary target illumination does not exist in anillumination dataset.

Referring to FIG. 8A, a nearest neighbors interpolation method may beused to select an illumination from an illumination dataset, based on aK-number of illuminations which are nearest to an input illuminationL_(u) in the illumination dataset. For example, when K is set to 3,three illuminations L_(1u), L_(2u), and L_(3u) which are nearest to theinput illumination Lu, may be identified from the illumination dataset.Based on the three nearest illuminations L_(1u), L_(2u), and L_(3u),three color transforms T_(1u), T_(2u), and T_(3u) are computed, using alinear or non-linear transform estimation method as shown in FIG. 7A.The three color transforms T_(1u), T_(2u), and T_(3u) may map thenearest illuminations L_(1u), L_(2u), and L_(3u) to a referenceillumination L_(d), respectively.

A first color transform T(L_(u)→L_(d)) that maps the input illuminationL_(u) to the reference illumination L_(d) may be computed as a weightedaverage of the three color transforms T_(1u), T_(2u), and T_(3u) asfollows: T(L_(u)→L_(d))=w_(1u)*T_(1u)+w_(2u)*T_(2u)+w_(3u)*T_(3u),wherein w_(1u), w_(2u), and w_(3u) are weights having values in a rangefrom 0 to 1.

Referring to FIG. 8B, a nearest neighbors interpolation method may beused to select an illumination from the illumination dataset, based on aK-number of illuminations which are nearest to an output illuminationL_(v) in the illumination dataset. For example, when K is set to 3,three illuminations L_(1v), L_(2v), and L_(3v) which are nearest to theoutput illumination L_(v), may be identified from the illuminationdataset. Based on the three nearest illuminations L_(1v), L_(2v), andL_(3v), three color transforms T_(1v), T_(2v), and T_(3v) are computed,using a linear or non-linear transform estimation method as shown inFIG. 7B. The three color transforms T_(1v), T_(2v), and T_(3v) may mapthe reference illumination L_(d) to the nearest illuminations L_(1v),Lv_(u), and L_(3v), respectively.

A second color transform T(L_(d)→L_(v)) that maps the referenceillumination L_(d) to the reference illumination may be computed as aweighted average of the three color transforms T_(1v), T_(2v), andT_(3v) as follows:T(L_(d)→L_(v))=w_(1v)*T_(1v)+w_(2v)*T_(2v)+w_(3v)*T_(3v), whereinw_(1v), w_(2v), and w_(3v) are weights having values in a range from 0to 1.

The first color transform and the second color transform that areacquired via the methods shown in FIGS. 8A and 8B may be applied to aninput image, for example, by the color transformation module 230 of FIG.1 , or may be used as a first ground-truth color transform and a secondground-truth color transform to train the first neural network 40 andthe second neural network 50 of the color transform estimation module220 of FIG. 1 .

FIG. 9 illustrates a method of applying a first color transform and asecond color transform to an input image according to embodiments of thepresent disclosure.

Referring to FIG. 9 , once a first color transform that maps an inputillumination to a reference illumination, and a second color transformthat maps the reference illumination to a target illumination areobtained, the first color transform is applied to an input image whichis captured under the input illumination in operation S311, to obtain anintermediate image under the reference illumination. In operation S312,the second color transform is applied to the intermediate image toobtain an output image under the target illumination.

FIG. 10 illustrates a result of converting an input illumination to areference illumination and converting the reference illumination to atarget illumination according to embodiments of the present disclosure.

As shown in FIG. 10 , an arbitrary input illumination is converted intoan arbitrary target illumination using a predetermined illumination as amiddle proxy between the arbitrary input illumination and the arbitrarytarget illumination. For example, a standard daylight illumination maybe used as the middle proxy to provide accurate color transforms sincethe daylight illumination has a wide spectral distribution and underwhich most colors are well represented.

However, embodiments of the present disclosure are not limited to usinga middle proxy between an input illumination and a target illumination,and may cover a method of directly mapping an input illumination to atarget illumination.

FIG. 11 illustrates a method of converting an input image under an inputillumination directly to an output image under a target illuminationwithout using a middle proxy, such as a daylight illumination.

Referring to FIG. 11 , in operation S411, an input image is obtained andan input illumination L_(u) of the input image is identified using anauto-white-balance (AWB) module. In operation S412, a color transformT(L_(u)→L_(v)) that maps the input image from the input illuminationL_(u) to a target illumination L_(v) is computed using a linear ornon-linear transform estimation method or a neural network-basedtransform estimation method. The target illumination L_(v) may beidentified from a user input. In operation S413, the color transform isapplied to the input image to re-illuminate the input image under thetarget illumination L_(v) to obtain an output image under the targetillumination L_(v).

FIG. 12 illustrates a method of training a neural network-basedtransform estimator according to embodiments of the present disclosure.

Referring to FIG. 12 , a transform estimator may include at least oneneural network which is trained to perform operation S412 of FIG. 11 .

The transform estimator may be implemented as a multi-layer perceptron(MLP), and may have a network structure as shown in FIG. 6A or 6B. Thetransform estimator may receive a pair of an input illumination and atarget illumination (L_(i), L_(j)) as input of the transform estimator,and output an estimated color transform T(L_(i)→L_(j)). A differencebetween the estimated color transform and a ground-truth color transformmay be computed as a loss of the transform estimator, and the transformestimator may be trained until the loss reaches a predetermined minimumvalue, or converges into a constant value with a preset margin.

FIG. 13 illustrates a use application of a color transformation methodaccording to embodiments of the present disclosure.

Referring to FIG. 13 , once a target illumination is input, at least onecolor transform that converts an input image from an input illuminationto the target illumination is estimated, and the estimated at least onecolor transform is applied to the input image to generate an augmentedimage under the target illumination. The input image (and/or thehistogram) and the augmented image (and/or the augmented histogram) arestored in an image database, to be used as a training dataset for neuralnetworks.

FIG. 14 is a diagram of devices for performing color transformestimation according to embodiments of the present disclosure. FIG. 14includes a user device 1100, a server 1120, and a network 1130. The userdevice 1110 and the server 1120 may interconnect via wired connections,wireless connections, or a combination of wired and wirelessconnections. An electronic device illustrated in FIG. 1 may correspondto the user device 1100 or a combination of the user device 1100 and theserver 1120. For example, all or at least a part of the processor 200illustrated in FIG. 1 may be included in the server 1120, and the restof the elements illustrated in FIG. 1 may be included in the user device1100.

The user device 1100 includes one or more devices configured to generatean output image. For example, the user device 1100 may include acomputing device (e.g., a desktop computer, a laptop computer, a tabletcomputer, a handheld computer, a smart speaker, a server, etc.), amobile phone (e.g., a smart phone, a radiotelephone, etc.), a cameradevice (e.g., a camera 100 illustrated in FIG. 1 ), a wearable device(e.g., a pair of smart glasses or a smart watch), or a similar device.

The server 1120 includes one or more devices configured to receive animage and perform an AI-based image processing on the image to obtain acolor-transformed image, according to a request from the user device1100.

The network 1130 includes one or more wired and/or wireless networks.For example, network 1130 may include a cellular network (e.g., a fifthgeneration (5G) network, a long-term evolution (LTE) network, a thirdgeneration (3G) network, a code division multiple access (CDMA) network,etc.), a public land mobile network (PLMN), a local area network (LAN),a wide area network (WAN), a metropolitan area network (MAN), atelephone network (e.g., the Public Switched Telephone Network (PSTN)),a private network, an ad hoc network, an intranet, the Internet, a fiberoptic-based network, or the like, and/or a combination of these or othertypes of networks.

The number and arrangement of devices and networks shown in FIG. 14 areprovided as an example. In practice, there may be additional devicesand/or networks, fewer devices and/or networks, different devices and/ornetworks, or differently arranged devices and/or networks than thoseshown in FIG. 14 . Furthermore, two or more devices shown in FIG. 14 maybe implemented within a single device, or a single device shown in FIG.14 may be implemented as multiple, distributed devices. Additionally, oralternatively, a set of devices (e.g., one or more devices) may performone or more functions described as being performed by another set ofdevices.

FIG. 15 is a diagram of components of one or more devices of FIG. 14according to embodiments of the present disclosure. An electronic device2000 may correspond to the user device 1100 and/or the server 1120.

The electronic device 2000 includes a bus 2010, a processor 2020, amemory 2030, an interface 2040, and a display 2050.

The bus 2010 includes a circuit for connecting the components 2020 to2050 with one another. The bus 2010 functions as a communication systemfor transferring data between the components 2020 to 2050 or betweenelectronic devices.

The processor 2020 includes one or more of a central processing unit(CPU), a graphics processor unit (GPU), an accelerated processing unit(APU), a many integrated core (MIC), a field-programmable gate array(FPGA), a digital signal processor (DSP), a machine learningaccelerator, a neural processing unit (NPU). The processor 2020 may be asingle core processor or a multi core processor. The processor 2020 isable to perform control of any one or any combination of the othercomponents of the electronic device 2000, and/or perform an operation ordata processing relating to communication. For example, the processor2020 may include all or at least a part of the elements of the processor200 illustrated in FIG. 1 . The processor 2020 executes one or moreprograms stored in the memory 2030.

The memory 2030 may include a volatile and/or non-volatile memory. Thememory 2030 stores information, such as one or more of commands, data,programs (one or more instructions), applications 2034, etc., which arerelated to at least one other component of the electronic device 2000and for driving and controlling the electronic device 2000. For example,commands and/or data may formulate an operating system (OS) 2032.Information stored in the memory 2030 may be executed by the processor2020. In particular, the memory 2030 may store original images andprocessed images (e.g., color transformed images).

The applications 2034 include the above-discussed embodiments. Inparticular, the applications 2034 may include programs to execute theauto-white-balance module 210, the color transform estimation module220, and the color transformation module 230 of FIG. 1 , and to performoperations S211-S217 of FIG. 2 . These functions can be performed by asingle application or by multiple applications that each carry out oneor more of these functions. For example, the applications 2034 mayinclude a photo editing application. When the photo editing applicationreceives a user request to convert colors of an image to colors under atarget illumination, the photo editing application may identify an inputillumination of the image using an auto-white-balance module, estimateat least one color transform, and apply the at least one color transformto the input image to transform the color of the image to the colorsunder the target illumination. The photo editing application may displayand store the color transformed image.

The display 2050 includes, for example, a liquid crystal display (LCD),a light emitting diode (LED) display, an organic light emitting diode(OLED) display, a quantum-dot light emitting diode (QLED) display, amicroelectromechanical systems (MEMS) display, or an electronic paperdisplay. The display 2050 can also be a depth-aware display, such as amulti-focal display. The display 2050 is able to present, for example,various contents, such as text, images, videos, icons, and symbols.

The interface 2040 includes input/output (I/O) interface 2042,communication interface 2044, and/or one or more sensors 2046. The I/Ointerface 2042 serves as an interface that can, for example, transfercommands and/or data between a user and/or other external devices andother component(s) of the electronic device 2000.

The communication interface 2044 may enable communication between theelectronic device 2000 and other external devices, via a wiredconnection, a wireless connection, or a combination of wired andwireless connections. The communication interface 2044 may permit theelectronic device 2000 to receive information from another device and/orprovide information to another device. For example, the communicationinterface 2044 may include an Ethernet interface, an optical interface,a coaxial interface, an infrared interface, a radio frequency (RF)interface, a universal serial bus (USB) interface, a Wi-Fi interface, acellular network interface, or the like. The communication interface2044 may receive or transmit a raw image, a processed image, and atarget illumination from or to an external device.

The sensor(s) 2046 of the interface 2040 can meter a physical quantityor detect an activation state of the electronic device 2000 and convertmetered or detected information into an electrical signal. For example,the sensor(s) 2046 can include one or more cameras (e.g., a camera 100illustrated in FIG. 1 ) or other imaging sensors for capturing images ofscenes. The sensor(s) 2046 can also include any one or any combinationof a microphone, a keyboard, a mouse, and one or more buttons for touchinput. The sensor(s) 2046 can further include an inertial measurementunit. In addition, the sensor(s) 2046 can include a control circuit forcontrolling at least one of the sensors included herein. Any of thesesensor(s) 2046 can be located within or coupled to the electronic device2000.

The color transformation method may be written as computer-executableprograms or instructions that may be stored in a medium.

The medium may continuously store the computer-executable programs orinstructions, or temporarily store the computer-executable programs orinstructions for execution or downloading. Also, the medium may be anyone of various recording media or storage media in which a single pieceor plurality of pieces of hardware are combined, and the medium is notlimited to a medium directly connected to an electronic device, but maybe distributed on a network. Examples of the medium include magneticmedia, such as a hard disk, a floppy disk, and a magnetic tape, opticalrecording media, such as CD-ROM and DVD, magneto-optical media such as afloptical disk, and ROM, RAM, and a flash memory, which are configuredto store program instructions. Other examples of the medium includerecording media and storage media managed by application storesdistributing applications or by websites, servers, and the likesupplying or distributing other various types of software.

The color transformation method may be provided in a form ofdownloadable software. A computer program product may include a product(for example, a downloadable application) in a form of a softwareprogram electronically distributed through a manufacturer or anelectronic market. For electronic distribution, at least a part of thesoftware program may be stored in a storage medium or may be temporarilygenerated. In this case, the storage medium may be a server or a storagemedium of a server.

A model related to the neural networks described above may beimplemented via a software module. When the model is implemented via asoftware module (for example, a program module including instructions),the model may be stored in a computer-readable recording medium.

Also, the model may be a part of the electronic device described aboveby being integrated in a form of a hardware chip. For example, the modelmay be manufactured in a form of a dedicated hardware chip forartificial intelligence, or may be manufactured as a part of an existinggeneral-purpose processor (for example, a CPU or application processor)or a graphic-dedicated processor (for example a GPU).

While the embodiments of the disclosure have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope as defined by thefollowing claims.

What is claimed is:
 1. An electronic device for processing image data,the electronic device comprising: a user interface configured to receivea target illumination; at least memory storing instructions andconfigured to store an illumination dataset of raw sensor images thatare captured under a plurality of different illuminations including areference illumination; and at least one processor configured to executethe instructions to: obtain a raw input image that is captured under aninput illumination; obtain an intermediate image having colors capturedunder the reference illumination, from the raw input image, based on afirst color transform that maps the input illumination to the referenceillumination in the illumination dataset; and obtain an output imagehaving colors captured under the target illumination, from theintermediate image, based on a second color transform that maps thereference illumination in the illumination dataset, to the targetillumination
 2. The electronic device of claim 1, wherein the at leastone processor is further configured to perform illumination estimationbased on the raw input image transformed by the first color transformand the second color transform, via an auto-white balance module.
 3. Theelectronic device of claim 1, wherein the at least one processor isfurther configured to create an augmented data set including the outputimage, and input the augmented data set to an artificial intelligence(AI)-based image processing model to train the AI-based image processingmodel.
 4. The electronic device of claim 1, wherein the raw sensorimages and the raw input image are unprocessed Bayer images or imageswhich are not corrected via an image signal processor (ISP).
 5. Theelectronic device of claim 1, wherein the first color transform and thesecond color transform are represented as a first transformation matrixand a second transformation matrix, respectively, and wherein the atleast one processor is further configured to: obtain the first colortransform via a first neural network configured to receive a ratio ofRGB values of the input illumination and output elements of the firsttransformation matrix; and obtain the second color transform via asecond neural network configured to receive a ratio of RGB values of thetarget illumination and output elements of the second transformationmatrix.
 6. The electronic device of claim 5, wherein the first neuralnetwork and the second neural network are trained using a training inputillumination, a training target illumination, and the referenceillumination that are obtained from the illumination dataset, andwherein the training input illumination, the training targetillumination, and the reference illumination are obtained from anachromatic patch in a color rendition chart of the raw sensor images. 7.The electronic device of claim 1, wherein the at least one processor isfurther configured to: based on the illumination dataset not includingthe input illumination, identify a K number of illuminations that arenearest to the input illumination in the illumination dataset, and use aweighted sum of color transforms of the K number of illuminations as thefirst color transform, and wherein K denotes a natural number that isgreater than or equal to
 2. 8. The electronic device of claim 1, whereinthe at least one processor is further configured to: based on theillumination dataset not including the target illumination, identify a Knumber of illuminations that are nearest to the target illumination inthe illumination dataset, and use a weighted sum of color transforms ofthe K number of illuminations as the second color transform, and whereinK denotes a natural number that is greater than or equal to
 2. 9. Theelectronic device of claim 1, further comprising: a camera configured tocapture the raw input image; and a display, wherein the user interfaceis further configured to receive, as a user input, the targetillumination and a request for creating a synthesized image of the rawinput image, and wherein the at least one processor is furtherconfigured to control the display to display the output image as thesynthesized image of the raw input image that is re-illuminated underthe target illumination.
 10. A method for processing image data, themethod comprising: obtaining a raw input image that is captured under aninput illumination; obtaining a target illumination from a user input;obtaining an intermediate image having colors captured under a referenceillumination, from the raw input image, based on a first color transformthat maps the input illumination to the reference illumination in anillumination dataset of raw sensor images that are captured under aplurality of different illuminations; and obtaining an output imagehaving colors captured under the target illumination, from theintermediate image, based on a second color transform that maps thereference illumination in the illumination dataset to the targetillumination.
 11. The method of claim 10, further comprising: performingillumination estimation on the raw input image transformed by the firstcolor transform and the second color transform via an auto-white balancemodule.
 12. The method of claim 10, further comprising: creating anaugmented data set including the output image, and input the augmenteddata set to an artificial intelligence (AI)-based image processing modelto train the AI-based image processing model.
 13. The method of claim10, wherein the raw input image is an unprocessed Bayer image or animage which is not corrected via an image signal processor (ISP). 14.The method of claim 10, wherein the first color transform and the secondcolor transform are represented as a first transformation matrix and asecond transformation matrix, respectively, and wherein the methodfurther comprises: obtaining the first color transform via a firstneural network configured to receive a ratio of RGB values of the inputillumination and output elements of the first transformation matrix; andobtaining the second color transform via a second neural networkconfigured to receive a ratio of RGB values of the target illuminationand output elements of the second transformation matrix.
 15. The methodof claim 14, further comprising: training the first neural network usinga training input illumination and the reference illumination; andtraining the second neural network using the reference illumination anda training target illumination, wherein the training input illumination,the training target illumination, and the reference illumination areobtained from the illumination dataset.
 16. The method of claim 10,further comprising: based on the illumination dataset not including theinput illumination, identifying a K number of illuminations that arenearest to the input illumination in the illumination dataset, and usinga weighted sum of color transforms of the K number of illuminations asthe first color transform, wherein K denotes a natural number that isgreater than or equal to
 2. 17. The method of claim 10, furthercomprising: based on the illumination dataset not including the targetillumination, identifying a K number of illuminations that are nearestto the target illumination in the illumination dataset, and using aweighted sum of color transforms of the K number of illuminations as thesecond color transform, wherein K denotes a natural number that isgreater than or equal to
 2. 18. A non-transitory computer-readablestorage medium storing a program that is executable by at least oneprocessor to perform a method for processing image data, the methodcomprising: obtaining a raw input image that is captured under an inputillumination; obtaining a target illumination from a user input;obtaining an intermediate image having colors captured under a referenceillumination, from the raw input image, based on a first color transformthat maps the input illumination to the reference illumination in anillumination dataset of raw sensor images that are captured under aplurality of different illuminations; and obtaining an output imagehaving colors captured under the target illumination, from theintermediate image, based on a second color transform that maps thereference illumination in the illumination dataset to the targetillumination.
 19. The non-transitory computer-readable storage medium ofclaim 18, wherein the method further comprises: obtaining the firstcolor transform via a first neural network configured to receive a ratioof RGB values of the input illumination and output elements of a firsttransformation matrix that represents the first color transform; andobtaining the second color transform via a second neural networkconfigured to receive a ratio of RGB values of the target illuminationand output elements of a second transformation matrix that representsthe second color transform.
 20. The non-transitory computer-readablestorage medium of claim 18, wherein the method further comprises: basedon the illumination dataset not including the input illumination,identify a K number of first illuminations that are nearest to the inputillumination in the illumination dataset, and use a weighted sum ofcolor transforms of the K number of first illuminations as the firstcolor transform; and based on the illumination dataset not including thetarget illumination, identify a K number of second illuminations thatare nearest to the target illumination in the illumination dataset, anduse a weighted sum of color transforms of the K number of secondilluminations as the second color transform, wherein K denotes a naturalnumber that is greater than or equal to 2.